Enhanced video system

ABSTRACT

A method and system for enhancement of video systems using wireless device proximity detection. The enhanced video system consists of one or more video capture devices along with one or more sensors detecting the presence of devices with some form of wireless communications enabled. The proximity of a device communicating wirelessly is sensed and cross referenced with received video image information. Through time, movement of wirelessly communicating mobile devices through a venue or set of venues can be deduced and additionally cross referenced to and augmented over image data from the set of video capture devices.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/640,209 (filed Jun. 30, 2017), which claims priority from provisionalU.S. patent application Ser. No. 62/382,246 (filed Aug. 31, 2016). Thisapplication is a continuation-in-part of, and claims the priority of,U.S. patent application Ser. No. 14/556,034 (filed Nov. 28, 2014). Theentire content of each aforementioned patent filing is herebyincorporated by reference.

BACKGROUND OF INVENTION Field of Invention

The present invention relates to the use of proximity detection andrecognition of mobile communications devices such as smart phones toenhance video systems consisting of cameras and other video or imagecapture devices along with video or image storage, analysis, and displaysubsystems.

Description of Related Art

Video systems are widely deployed for surveillance purposes by venueoperators. Such systems use a plurality of video camera units, acommunication network to transmit the video data and other related datasuch as motion detector data; a data storing terminal for receiving andstoring the video data from the video camera units through the network;a display terminal, for displaying the video data from the data storingterminal and the video camera units; and one or more control servers.

Understanding visitor behavior is critical to the design, operation andoptimization of public spaces. Visitor behavior information is valuableto both owner/operator of the space, venue or real estate as well as thesecurity personnel and merchant staff operating therein. Newergeneration video surveillance systems make use of image analysis toautomatically detect and alert on certain conditions relevant to thesystem operator for the purposes, for example, of physical security.Such image analysis and condition detection include, for example,detection of a person loitering in a single camera's field of view.Given a single video capture device or camera has a limited field ofview such analysis is limited.

Detection of the path of a moving object across the combined field ofvision from multiple single video capture devices or cameras is acurrent area of active research. Such systems require substantialcalibration as well as image analysis resources and are subject toerror.

Adoption of mobile devices including mobile phones, smart phones, andtablets has enabled new means of understanding visitor interaction witha specific venue or set of venues.

Mobile devices are typically aware of their location thanks totechnologies such as the global positioning system (GPS) and can providethis information to both software applications as well as the mobilecommunication network itself. GPS requires the mobile device to have aclear view of the sky in order to receive positioning information frommultiple GPS satellites which are typically deployed above the equatorin geosynchronous orbit. Because of this, GPS does not work well indoorsor in outdoor locations that have obscured access to the portion of thesky where GPS satellites appear. This includes outdoor locations withtall buildings or other large infrastructure such as bridges (referredto as “urban canyons”) and areas with dense foliage.

Mobile communications networks also have extensive positioningcapabilities. Terrestrial based mobile communications network deploys alarge number of base stations. The design of mobile communicationsnetworks has the mobile device stay in constant association with one ormore base stations. As a result, the mobile communications network hasinformation about the macro location of a mobile device. The range of abase station can be several miles in diameter and accurate positioningis made difficult due to signal strength fluctuations and othertechnical challenges.

Newer systems such as “Assisted GPS” are designed to combine informationfrom GPS and mobile communications networks to improve accuracy. Thesesystems also suffer from accuracy problems when GPS coverage is lost forextended periods of time.

Alternatives to satellite based location systems are emerging. One suchexample involves frequently sensing and recording the identification(typically by MAC address) and the signal strength of all the 802.11based WiFi access points at a specific location. This recording istypically performed with a specially designed vehicle. When a mobiledevice needs to know its position, the mobile device itself can senseall the 802.11 access points in its vicinity and compare this with thepreviously recorded information to determine its location. This approachis effective in urban locations where GPS does not perform well.Increased AP (“Access Point”) density and frequent recording increasethe accuracy of this type of system. These kinds of systems also operateindependently of the mobile communications network. Once location isdetermined at the mobile device, it can be used by software applicationson the mobile device to, for example, display location on a mapbackground or it can be reported to a central server via the Internet.

Various systems exist to furnish information to venue operators relatedto visitor behavior in retail and public spaces. These include thermalcameras, stereoscopic video cameras, and infrared light beams as well asother more application specific technologies such as road inductionloops.

Such systems lack the ability to accurately detect and report onbehavior of visitors between visits to a venue. This is an active areaof innovation. Innovations in camera technology including facialrecognition are being actively pursued by several parties.

To improve venue operator understanding of the behavior of visitors totheir venue or venues, an improved system would be useful for enablingbetter understanding of the behavior of customers including visitfrequency, visit duration, visit path with mobile devices.

One solution is to provide a system for receiving informationtransmitted by wirelessly communicating mobile devices and inferringbehavior from this information.

This understanding of visitor path or movement through a venue can becombined with video capture device or camera based video surveillancesystems. This combination provides for more effective use of such cameraor video based venue surveillance systems.

SUMMARY OF THE INVENTION

The present invention integrates proximity recognition of wirelessenabled mobile devices and associated mobile device user movement withvideo systems (typical used for surveillance).

In accordance with the present invention, there is provided a system andprocess which enables a venue operator to understand the behavior ofvisitors who carry wirelessly communicating mobile devices incombination with video systems consisting of one or more video and/orimage capture devices.

Accordingly, the present invention involves one or more proximityrecognition devices (PROs) operating at a venue. Electromagneticcommunications interactions between wireless communicating mobiledevices and related wireless communications infrastructure are recordedby the proximity recognition device PRD, analyzed and sent to thecentral controller. Knowledge of interactions with mobile devicesprovides the proximity recognition system (PRS) with the ability todetect presence and specific location of the mobile device (i.e. itsassociated visitor) within the venue. As visitors move through thevenue, the proximity recognition system (PRS) combines, cross referencesand/or shares its understanding of visitor behavior with video system toenable more effective surveillance of the venue.

Accordingly, the present invention involves improvement of locationaccuracy. As location accuracy improves with the amount of interaction(between mobile devices and the system's PRD) to analyze, the PRD mayoptionally be tuned to prompt more or less interaction with the mobiledevice based on the objectives of the venue operator.

Accordingly, the present invention involves improvement of locationaccuracy through common trilateration and triangulation techniques oninteraction data received from three or more in PRDs in the venue. It isalso possible to determine location information based on a single PRD.This location information is obviously less accurate compared tolocation information that is based on multiple PRDs.

In one embodiment, the PRS uses PRO functionality commonly incorporatedinto wireless communications infrastructure devices including WiFiaccess points (APs). In another embodiment, the PRS makes uses ofdedicated PRS sensor devices. In another embodiment, the PRS makes useof a combination of existing wireless communications infrastructuredevices and dedicated PRD sensor devices. In another embodiment, the PRS(and/or PRD(s)) and the Video Capture Device (VCD) are incorporated inthe same device (with the result that their respective electromagneticinteractive field and visual field of view, are (at least, partially)co-extensive).

Accordingly, the present invention's central controller of the proximityrecognition system (PRS), in one embodiment, is designed to run as anInternet connected appliance providing a cloud based service.Alternative embodiments enable the central controller to be run by a bya third party providing new or existing merchant analytics serviceincluding “footfall” analytics and/or video based security services.When the customer enters a specific merchant venue, the systemrecognizes the event based on the detection of the visitor's wirelesslycommunicating mobile device. Visitor behavior such as path taken throughthe venue and visit duration is reported to the central controller forappropriately anonymized analysis by the venue's staff.

Accordingly, the present invention involves analysis by the centralcontroller of information received from a plurality of proximityrecognition devices PRDs deployed in venues connected to the centralcontroller by a communications network such as the Internet. Thisinformation is then cross referenced with received video imageinformation by the central controller to provide an enhanced videosystem (EVS). Results of this analysis are then transmitted to oravailable for display to venue operator or its staff, subcontractors, oragents at their request.

The cross referencing of video image data with mobile device proximityis based on a correlated geospatial understanding of the venue in whichthe EVS is operating. Given an EVS with a plurality of video capturedevices (VCDs) and a plurality of PRDs, the EVS can cross reference agiven VCD's field of view at a given instance in time with PRDs in thevicinity and, as a result, use information from these PRDs to determinethe one or more mobile devices expected to be within the given VCD'sfield of view at a given instance in time.

When focused on a given VCD's field of view at present, this correlatedunderstanding between the VCD's field of view and mobile devices in thegiven VCD's field of view enables the EVS to determine the previouslocations within the venue of the mobile devices in the VCD's field ofview at present time. Image data from those previous locations canoptionally be retrieved and displayed to an operator.

When focused on a given VCD's field of view at some given time in thepast, this correlated understanding between the VCD's field of view andmobile devices in given VCD's field of view enables the EVS to determinethe previous and future (with respect to the given time in the past)locations of the mobile devices in the VCD's field of view at the pasttime of interest.

Given two or more distinct fields of view from the same VCD atdifferent, specified times or two or more different VCDs at different,specified times, the EVS can determine the commonality of mobile devicesthat were in or in the vicinity of the given fields of view at thespecified times. This is useful for identifying one or more devices ofinterest (DoIs) whose behavior including path through the venue orseries of monitored venues can be examined.

When focused on a given VCD's field of view at some given time, the EVScan determine movement of multiple mobile devices having the sameapproximate path through the venue and/or current visit arrival timeand/or previous arrival times or any combination of similar facts todefine a group. Arrival and movement of groups could optionally producealerts for staff tasked with venue security and safety.

When focused on a given VCD's field of view at some given time, theimage display can be augmented to show behavioral characteristics ofeach mobile device and its presumed user in the given VCD's field ofview. This can include information about previous venue visit historyassociated with the mobile device, arrival time, dwell time at the venueduring this and/or previous visits, possible movement within a group ofpeople, group detail such as size of group as well as information aboutthe mobile device's path through the venue during this and/or previousvisits. Suggest augmented images could be filtered accordingly to userdefined criteria.

The correlated understanding between a VCD's field of view and mobiledevices expected to be within the proximity of the VCD's field of viewis bidirectional. In addition to the correlation of given VCD's field ofview at a specific instant in time to mobile devices in the proximity ofthe field of view, a specifically identified and detected mobile deviceor set of mobile devices can be correlated back to a set of imagesexpected to have the mobile device and its user in view at a specificinstant in time.

Given a set of previous and possibly relatively future locations of oneor more devices of interest (DoIs), the EVS can, then, in near real timecontinue to determine their present location and enable the display ofcurrent imagery from the one or more VCDs whose fields of view areexpected to contain the DoI(s) and their associated user. Optionally,imagery can be augmented to show the DoIs and their associated users.Such “find and follow” functionality is expected to be useful in variousvideo surveillance scenarios.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates the system architecture of the enhanced video system(EVS).

FIG. 2 illustrates the relationship between mobile devices and the fieldof view of a video capture device (VCD).

FIG. 3 illustrates the relationship between mobile devices detected bymultiple proximity detection devices (PRDs) and the field of view of avideo capture device (VCD).

FIG. 4 is a sample depiction of image information along with informationabout objects detected in the scene or practical field of view of a VCD.This object information may be generated by the VCD itself or someexternal device that has access to the series of images produced by theVCD.

FIG. 5 illustrates the relationship between mobile devices detected bymultiple proximity detection devices (PROs), the fields of view of amultiple video capture devices (VCDs).

FIG. 6 illustrates the relationship between an example mobile device asit moves through a venue and the EVS.

FIG. 7 is an architecture drawing of a multi venue distributed enhancedvideo system (EVS) including PRD units, VCD units and a centralcontroller.

FIG. 8 is a block diagram of the EVS central controller.

FIG. 9 is an alternative architecture of the EVS which separates thevideo management subsystem VMS and the PRS subsystems.

FIG. 10 is a logic diagram illustrating the process of a proximityrecognition device (PRD) operating in a venue.

FIG. 11 is a logic diagram illustrating the process of an EVS usinginformation received from associated proximity recognition devices(PRDs) operating in a venue in conjunction with image informationreceived from the VCDs deployed within the venue to calculation imageaugmentation data.

FIG. 12 is a logic diagram illustrating the process of an EVS using animage received from a VCD deployed in a venue and its implied field ofview at a specific time along with information received from PRDs in thesame venue to determine all of the mobile devices within the VCD's fieldof view at the specific time of interest as well as the movement andother behavioral information associated with the determined set ofmobile devices.

FIG. 13 is a logic diagram illustrating the process of an EVS using animage received from a VCD deployed in a venue and its associated fieldof view at a specific time along with other instances of images receivedfrom the same or different VCDs deployed in a venue and their associatedfields of view at different instances in time along with informationreceived from PRDs in the same venue to determine all of the mobiledevices expected to be in all or some subset of the supplied VCD fieldsof view at the specified associated times of interest as well as themovement and other behavioral information associated with the determinedset of mobile devices.

DETAILED DESCRIPTION OF THE INVENTION

In this section, the present invention is described in detail withregard to the drawing figures briefly described above.

For purposes of description the following terms have these meanings:

The terms “real estate provider”, “venue owner”, “venue operator”, “realestate operator” and “real estate owner” unless otherwise specifiedbelow are used interchangeably to refer to the entity that owns and/oroperates real estate space. Real estate providers in the context of thepresent invention are interested in one or both of the followingobjectives: understand behavior of visitors to their owned and/oroperated space and ensure the security of visitors, tenant staff, andoperations personnel within their owned and/or operated space.

The terms “venue”, “physical venue”, “premise”, “space”, “real estate”,and “real estate premise” unless otherwise specified below are usedinterchangeably to refer to a specific physical space owned and/oroperated by a real estate provider. Venues include malls, stores,parking facilities, shops, and theatres as well as other types of spacesincluding hotels, motels, inns, airports, warehouses, dock facilities,arenas, hospitals, schools, colleges, universities, corporate campuses,libraries, galleries, stations, parks, parking lots, and stadiums. Inalternate embodiments of the invention, space may include roadways onwhich vehicles operate.

The terms “WiFi, “Wifi”, “WLAN”, “Wireless Fidelity”, and “wirelesslocal area network” all refer to communications between mobile devicesand infrastructure elements (commonly referred to as “access points” orAPs). WLAN refers to devices and infrastructure using some variant ofthe 802.11 protocol defined by the institute of Electrical andElectronics Engineers (IEEE) or some future derivation.

The terms “video system”, “video management system”, “video securitysystem”, and “physical security system” all refer to systems deployedwith one or more video capture devices, video storage either at thevenue or in a central controller or some combination, and video displayalong with optional systems components for video analysis, accesscontrol, and other sensor technology such as sensor technology for thepurposes of venue surveillance and related security.

The terms “mobile device”, “wireless device”, “wirelessly communicatingmobile devices”, and “wireless enabled device”, all refer to devicesequipped to communicate over a wireless communications network includingwireless communications networks using the 802.11 protocol or otherwireless communications technologies such as LTE, WiMax, Bluetooth, or“5G”.

The terms “visitor”, “guest”, “user”, or “invitee” unless otherwisespecified below, are used interchangeably to refer to any party thatvisits a venue.

The term “image information”, “video information”, “video image data”,“image data”, “camera data”, “video frames”, and “video frame data”unless otherwise specified below, are used interchangeably to refer tothe series of images of a scene captured by a video capture device (VCD)often referred to as a camera and sent to the EVS central controller.These VCD images have one or more timestamps indicating when the imagewas captured by the VCD and, optionally, when the image was processed bythe EVS in some way. These VCD images also have an implied or explicitlydefined field of view which is considered to be part of overall imageinformation. This field of view information may be static or dynamic inthe case where the VCD can be panned, tilted or zoomed, Optionally,image information can include additional information about the sceneand/or objects detected in the scene. This optional scene or objectinformation may be generated by the VCD itself or some external devicethat has access to the series of images produced by the VCD.

The term “field of view”, “VCD field of view”, and “camera field ofview” unless otherwise specified below, are used interchangeably torefer to the fixed or possibly dynamic area that a VCD or camera devicehas visibility to at any given instant in time. It is noted that thefield of view of a VCD or camera has a theoretical maximum shape inthree dimensions (“theoretical field of view”) but is subject to bothshort term obstructions such as lighting conditions and objects such aspeople moving dynamically in the scene visible in the VCD or camera'sfield of view as well as semi-permanent obstructions from objects suchas signs and walls which may have been in the VCD's field of view uponinstallation or which may have been placed in the VCD's field of viewafter VCD installation. It is noted that obstruction of a VCD field ofview can impair the VCD's usefulness. It is further noted that PRDs andthe associated PRS are not subject to such field of view obstructionsand can detect mobile devices in the theoretical field of view butobstructed from practical VCD view (“practical field of view”) as wellas mobile devices in the vicinity of a VCD field of view.

The above defined terms are used to describe the preferred embodiment ofthe present invention in reference to the attached drawing figures.Where appropriate, parts are referred to with reference numerals.

Referring to FIG. 1, the principal components used to implement thepresent invention are illustrated in a block diagram. A system andmethod is provided for enhanced video surveillance by combining videocapture devices along with wireless proximity detection capabilitieswhich recognize and classify wirelessly communicating mobile devices.The enhanced video system (EVS) receives and optionally analyzes videoimages while receiving and analyzing information regarding the presenceof wirelessly communications devices at one or more known locationsdepicted in FIG. 1 as 100. As video images are captured by one or morevideo capture devices depicted in FIG. 1 as 106 and 107 and transmittedto a central controller 112 via communications interface 108, theproximity of a wirelessly communicating mobile device 101 or pluralityof wirelessly communicating mobile devices 101, 102, 103 is sensed byexamining the signal information (e.g. signal strength) at one or moreproximity recognition devices PRDs 104, 105 when, for example, thewirelessly communicating mobile device 101 initiates a communicationsrequest. The threshold of the signal information to signify detectionand proximity, is adjustable by user to adjust the physical dimensionsof the detection field of the PRDs. An identifier of the wirelesslycommunicating mobile device may be provided in the communicationsrequest. Communication requests may be sensed by one or a plurality ofproximity recognition devices PRDs 104, 105. Information received by theproximity recognition devices PRDs is analyzed, summarized and sent tothe Central Controller via communications interface 108. Communicationsinterface 108 is comprised of some combination of cable modems, DSL,DS1, DS3, SONET, Ethernet, fiber optic, WiMax, WiFi 802.11 or otherwireless technology such as CDMA, GSM or long term evolution (LTE) orother future communications capability to a communications network 110such as the Internet. Image information received by the video capturedevices (VCDs) 106, 107 is optionally stored for later retrieval,optionally analyzed, and sent to the Central Controller 112 viacommunications interface 108. The Central Controller 112 receivedinformation from VCDs 106,107 and PRDs 104, 105 using communicationsinterface 111 to a communications network 110. Image information fromthe VCDs and the PRDs may be transported over the same communicationsinterface 108 and network 110 or in an alternative embodiment overseparate communications infrastructure.

In the present context, a computer usable medium or computer readablemedium may be any medium that can contain or store the program for useby or in connection with the instruction execution system, apparatus ordevice. For example, the computer readable storage medium or computerusable medium may be, but is not limited to, a random access memory(RAM), read-only memory (ROM), or a persistent store, such as a massstorage device, hard drives, CDROM, DVDROM, solid state drives, tape,erasable programmable read-only memory (EPROM or flash memory), or anymagnetic, electromagnetic, infrared, optical, or electrical system,apparatus or device for storing information. Alternatively oradditionally, the computer readable storage medium or computer usablemedium may be any combination of these devices or even paper or anothersuitable medium upon which the program code is printed, as the programcode can be electronically captured, via, for instance, optical scanningof the paper or other medium, then compiled, interpreted, or otherwiseprocessed in a suitable manner, if necessary, and then stored in acomputer memory.

Applications, software programs or computer readable instructions may bereferred to as components or modules. Applications may be hard coded inhardware or take the form of software executing on a general purposecomputer such that when the software is loaded into and/or executed bythe computer, the computer becomes an apparatus for practicing theinvention, or they are available via a web service. Applications mayalso be downloaded in whole or in part through the use of variousdevelopment tools which enable the creation, implementation of thepresent invention. In this specification, these implementations, or anyother form that the invention may take, may be referred to astechniques. In general, the order of the steps of disclosed processesmay be altered within the scope of the invention.

Various embodiments of EVS are possible. VCD and PRD functionality makebe implemented in separate devices installed and operating in the venue.VCD and PRO functionality may be implemented in a single device typeinstalled and operating in the venue. As well, hybrid embodimentsincluding separate and combined device types are possible.

With reference to FIG. 2, a block diagram of illustrating therelationship between mobile devices and the field of view of a videocapture device (VCD).

Within a typical deployment venue environment 200, one or more VCDs 201and one or more PRDs 206 may be connected to the venue's existingcommunications network 207. In various embodiments, PRDs 206 and VCDs201 may be connected to the venue's communications network 207 via sometype of wired connection such as Ethernet 207 or wirelessly such as WLANnetwork 407 in the venue. Venue's communication network 207 is attachedby a communications interface such as DSL or cable modem 208 to widearea communications network 209 such as the Internet.

Within the venue, the VCDs record and transmit for optional storageimages with a given field of view at a given frequency. In some cases,image recording may be triggered by an external event such as motiondetection.

The VCDs send image Information on an event or periodic basis to thecentral controller 211 via communications network 209 and associatedcommunications interface 210. In this described embodiment,communications network 209 coupled to the venue's communications network207 by communications interface 208.

Within the venue, a plurality of mobile devices 202, 203, 204 isexpected to arrive and depart to and from the venue in a random nature.The PRD 206 or PRDs observes communications signals from these mobiledevices in its vicinity.

At any given time, these zero or more of these mobile devices 202, 203,204 present at the venue and their associated users may be within thefield of view 205 of a VCD 201 deployed in the venue. In the particularscenario depicted, mobile devices 204 and 203 are within the field ofview 205 of a VCD 201 deployed in the venue. It is possible that mobiledevice 202 is known to be in the venue (e.g. by the electromagneticcommunications interactions sensed by PRS/PRDs) but is not in the fieldof view of a VCD deployed in the venue.

The PRDs 206 also reports information regarding mobile deviceobservations on an event or periodic basis to the central controller 211of the enhanced video system via communications network 209 which inthis described embodiment is coupled to the venue's communicationsnetwork 207 by communications interface 208.

The Central Controller 211 can cross reference the field of view of agiven VCD 201 with mobile device location information received from PRO206 to determine the list of mobile devices expected to be in the fieldof view VCD 201 at a specific instance in time. The Central Controller211 has a geospatial understanding of both the VCD 201 field of view 205as well as the geospatial capability to determine the intersection of agiven VCD field of view with mobile device location information receivedfrom the proximity recognition devices PROs 206 and their associated PRSsubsystem. This geospatial understanding may, in one embodiment, use aglobal standard for geospatial location such as GPS uses or may use alocal or venue specific, in another embodiment, of locations within thevenue. In alternative embodiments, the field of view as well as mobiledevice location may be represented in three dimensions. In alternativeembodiments, VCD field of view and calculated mobile device locationscan be represented in two dimensions with an optional specification ofthe floor number if the venue has multiple floors.

In one embodiment, VCD fields of view are manually defined as twodimension polygons as an overlay of a physical venue's floorplan. Thelocations of the PRDs are also defined with respect to the same physicalfloorplan. This coordinated geospatial understanding of the venueenables EVS to determine the mobile devices within a given VCD's fieldof view at a given time and to, additionally, determine the zero or moreviews for a given mobile device's location in a venue at a given timeusing techniques well known to those skilled in the areas oftwo-dimensional geometry and computer science.

In an alternative implementation, the VCD field of view can be computedin three dimensional space based on the VCD's location, the threedimensional orientation of the camera (i.e. the angle that it ispointing at) and specification or assumption of the shape of the VCD'sfield of view. Given a coordinated geospatial understanding with thePRDs deployed in a given venue, the EVS can cross reference between agiven VCD field of view and the mobile devices expected to be in thatVCD's field of view at a given instant in time as well as to crossreference from a mobile device's location at a specific instant in timeto the zero or more VCD fields of view that the mobile device and itsuser are expected to appear in. Various mathematically techniques existfor determination overlap between three dimensional shapes such as aVCD's three dimensional field of view and mobile device location at agiven instant in time. These mathematical techniques include theSutherland-Hodgman algorithm as well as the algorithm developed byTeixeira and Creus.

In alternative implementations, the VCD's three-dimensional position andfield of view can be manually defined in the EVS or, alternatively, canbe sensed by the VCD itself using additional sensor technology such asGPS and gyroscope devices such as the AGD1 2022 FP6AQ gyroscope devicedesigned by STM Electronics used in the iPhone 4 smartphone developed byApple Inc. of Cupertino, Calif. In this alternative embodiment where theVCD senses its position and orientation, this information would besupplied to the EVS central controller 211 to enable it to crossreference between VCD image information specifically a frame of videoimagery received by the EVS central controller 211 from the VCD 201 andmobile devices 203 204 in this particular VCD's field of view at aspecific instant in time.

Additionally, the zero or more VCD fields of view can be determinedgiven a mobile device location sensed by the one or more PRDs in thevenue at specific instant in time. Once these fields of view aredetermined, the associated VCDs can be determined and the associatedimage information from the zero or more VCDs can be displayed from theassociated instant in time showing the user of the associated mobiledevice. This can be done repeated for every known location of the samemobile device to visually depict the path of the mobile device throughthe venue from its arrival to its departure. Additionally, imageinformation from a prior or future visit of the mobile device and itsassociated user can be shown for each time a location is sensed for agiven mobile device by the one or more PROs in the venue by determiningthe zero or more associated VCD fields of view the mobile device issensed and determined to be in and then showing the image informationfor the associated zero or more VCDs at the associated instant in time.

The PRS functionality of the EVS central controller 211 and the one ormore PRDs 206 deployed at the venue 200 can also be used to determine ifa given mobile device of interest remains at the venue 200 but is not inany of the practical fields of views of any of the VCDs 201 deployed atthe venue 200. This functionality is useful for determining if the userassociated with the mobile device of interest is in a theoretical fieldof view but is not evident in the image information received from any tothe VCDs 201 deployed in the venue 200 because of some form ofobstruction or is in the vicinity of a VCD 201 but not in the currentfield of view 205.

Given two or more distinct fields of view from the same VCD atdifferent, specified times or two or more different VCDs at different,specified times, the EVS can determine the a list of zero or more mobiledevices that were in or in the vicinity of the given fields of view atthe specified times. This knowledge can be used to identify one or moredevices of interest (DoIs) whose behavior including path through thevenue or series of monitored venues can be examined and alerted onaccording to some rules provided by a human operator or an externalsystem. In an alternative embodiment, knowledge path or movementinformation associated with one or more mobile devices can be used bythe EVS to augment video search and other video analytics functions.

With reference to FIG. 3, a block diagram of illustrating therelationship between mobile devices detected by multiple proximitydetection devices (PROs) and the field of view of a video capture device(VCD).

Within a typical deployment venue environment 300, one or more VCDs 301and one or more PRDs 306, 307, 208 may be connected to the venue'sexisting communications network 310.

Within the venue, the VCDs 301 record and transmit for optional storageimages with a given field of view at a given frequency.

Within the venue, a plurality of mobile devices 302, 303, 304 isexpected to arrive and depart to and from the venue in a random nature.The PRO 306, 307, 305 observes communications signals from these mobiledevices in their vicinity. When a mobile device communication isobserved by three or more PRDs, standard trilateration, triangulation,time of flight, or some combination of these or other locationcalculation techniques may be utilized to determine the location of themobile device with the venue.

At any given time, these zero or more of these mobile devices 302, 303,304 present at the venue and their associated users may be within thefield of view of a VCD 301 deployed in the venue. In the particularscenario depicted, mobile devices 304 and 303 are within the field ofview 308 of a VCD 301 deployed in the venue. Mobile device 302 is knownto be in the venue but is not in the field of view 308 of a VCD 301deployed in the venue.

Image information from the deployed VCDs 301 and mobile deviceinformation from the PROs 306, 307, 305 is communicated to CentralController 350 via communications network 330.

The Central Controller 350 cross references image and field of viewinformation from the VCDs 301 with the mobile device locationinformation as derived from information received from the PRDs 305, 306,307. In one embodiment, this cross referencing is accomplished bycomparing the three-dimensional location of the mobile devices 302, 303,304 with the three dimensional fields of view 308 of the one or moreVCDs 301 deployed at the venue using location reference informationstored in the Central Controller's VCD database 836 and PRD database832. Various mathematic techniques are well known for cross referencingand may be used to accomplish this intersection calculation task. Theseinclude polygonal intersection algorithms developed bySutherland-Hodgman and Teixeira and Creus In another embodiment, thiscross referencing is based on user-defined rules and parameters. In analternative embodiment, field of view 308 can be defined in the EVSusing PRS-related functions to cross reference between mobile devicessensed by PROs 305, 306, 307 that map into the field of view 308 whichis visible to VCD 301.

In one embodiment, the field of view of one VCD at a specific instant intime (may include previous X seconds) may be used to calculate themobile devices known to be visible to this VCD at this time. Once thisset or cohort of mobile devices of interest is determined, the cohortcan be reviewed to determine their current and past visit behaviorincluding association patterns with other mobile devices and theirassociated users.

With reference to FIG. 5, a block diagram illustrating the relationshipbetween mobile devices detected by multiple proximity detection devices(PRDs), the fields of view of a multiple video capture devices (VCDs)that may be installed in the venue.

Within a typical deployment venue environment 500, a plurality of PROdevices 506, 507, 508 and a plurality of VCD devices 501, 502 may berequired. With reference to the specific embodiment example depicted inFIG. 5, PROs 506, 507, 508 and VCDs 501, 502 may be connected to thevenue's existing communications network 520. In various embodiments, theVCDs and PRDs may be connected to the venue's communications network 520via some type of wired connection such as Ethernet or wirelessly such asWLAN network. Venue's communication network 520 is attached by acommunications interface such as DSL or cable modem 530 to wide areacommunications network 540 such as the internet.

Within venue 500, a plurality of wirelessly communicating mobile devices503, 504, 505 are expected to arrive and depart to and from venue 500 ina random nature. The PRDs in the venue observe communication signalsfrom wirelessly communicating mobile devices in its vicinity including,in one embodiment, WLAN communications.

As the wirelessly communicating mobile devices 503, 504, 505 areexpected to arrive and depart to and from venue 500, they will movethrough the fields of view of zero or more VCDs 501, 502. Depending onthe deployment of the VCDs within venue, it is possible, at any giveninstant in time, a mobile device and its associated user could be within the field of view of zero, or one, or more than one VCD field ofview.

In the particular scenario depicted, mobile device 504 is within thefield of view 509 of a VCD 502 as well as the field of view 510 of VCD501. Mobile device 505 is in field of view 509 of a VCD 502 but not thefield of view 510 of VCD 501. Mobile device 503 is in field of view 510of a VCD 501 but not the field of view 509 of VCD 502.

The Central Controller 560 cross references image and field of viewinformation from the VCDs 502, 501 with the mobile device locationinformation as derived from information received from the PROs 506, 507,508. This cross referencing is accomplished by comparing the location ofthe mobile device 503, 504, 505 with the fields of view 509, 510 of theone or more VCDs 501, 502 deployed at the venue using location referenceinformation stored in the Central Controller's VCD database 836 and PROdatabase 832. Standard mathematic techniques such as theSutherland-Hodgman algorithm may be used to accomplish this task.

In one embodiment, the location of the mobile device may be used tocalculate the fields of view of the VCDs which would be expected tocontain the wirelessly communicating mobile device and its user. Forexample, once the location of mobile device 505 is determined for thedepicted specific instant in time, such cross referencing calculationswould indicate that fields of view 509 and 510 would both contain imagesof mobile device 505 and its associated user.

With reference to FIG. 6, a block diagram illustrating the relationshipbetween an example mobile device as it moves through a venue.

Within a typical deployment venue environment 600, a plurality of PRDdevices 604, 605, 606, 607, 608 and a plurality of VCD devices 601, 602,603 may be deployed. With reference to the specific embodiment exampledepicted in FIG. 6, PRDs 604, 605, 606, 607, 608 and VCDs 601, 602, 603may be connected to the venue's existing communications network 630.Venue's communication network 630 is attached by a communicationsinterface of some form 640 to wide area communications network 650 suchas the Internet.

An example mobile device 620 is detected by multiple proximity detectiondevices (PRDs) and passes through the field of view of a multiple videocapture devices (VCDs).

As the wirelessly communicating mobile devices 621, 620 are expected toarrive and depart to and from venue 600, they will move through thefields of view of zero or more VCDs 601, 602, 603. Depending on thedeployment of the VCDs within venue, it is possible, at any giveninstant in time, a mobile device and its associated user could be within the field of view of zero, or one, or more than one VCD field ofview.

At a specific instant in time denoted ‘t_(x)’ in the example depicted,mobile device 620 is in the field of view 609 of VCD 601 as is anotherexample mobile device 621. These are denoted in FIG. 6 as 620@t_(x) and621@t_(x). It should be noted that t_(x) may be at the present time ormay be some past instant in time. Both possibilities are part of thisinvention.

Mobile device 620's past and future interaction with the venue or andany associated venues can be determined. In the example depicted,through cross reference analysis by Central Controller 670, mobiledevice 620 can be determined to have been in vicinity of VCD 601 but notwithin its field of view 609 at some specific instant in time denoted‘t_(p)’. Through a similar process, the Central Controller 670, ca ndetermine that the same mobile device 620 at some specific instant intime denoted ‘t_(f)’ is in the field of view 611 of VCD 603.

With reference to FIG. 7, an architecture drawing of a multi venuedistributed enhanced video system (EVS) including PRD units, VCD unitsand a central controller is shown.

Central Controller 760 is connected to one or more venues 700, 710 bycommunications network 750 through communications interface 723.Communications interface 723 comprises one or some combination of cablemodems, DSL, DS1, DS3, Ethernet, fiber optic, or some other future wiredconnectivity as well as WiFi 802.11 or Long Term Evolution (LTE) or someother current or future wireless technology in a manner well known tothose skilled in the area of communications technology.

Within the exemplary venue 700, one or more video capture devices PRDs704, 705 are deployed in a manner designed to provide appropriatevisibility of required physical spaces within venue 700.

One or more proximity recognition devices PRDs 701, 702 are alsodeployed within venue 700 in a manner designed to provide appropriatedetection of wirelessly communicating mobile devices within venue 700.

Proximity recognition devices PROs 701, 702 within venue 700, as well asProximity recognition devices PRDs 711 and 712 within venue 710 areconnected to communications network 750 through communicationsinterfaces 721 and 722 respectively, as previously described. In oneembodiment, PRDs 701, 702, for example, may be coupled to thecommunications infrastructure of venue 700 and communicate tocommunications network 750 through the venue's primary and possible backup communications interfaces 721 through some communications gateway 703(with communications interface 722 and communications gateway 713performing corresponding functions for PROs 711, 713 within venue 710).

Video capture devices VCDs 704, 705 within venue 700, as well as videocapture devices PRDs 714 and 715 within venue 710 are connected tocommunications network 750 through communications interfaces 721 and 722respectively, as previously described. In one embodiment, VCDs 705, 704,for example, may be coupled to the communications infrastructure ofvenue 700 and communicate to communications network 750 through thevenue's primary and possible back up communications interfaces 721through some communications and/or application services gateway 703(with communications interface 722 and communications/applicationsservices gateway 713 performing corresponding functions for VCDs 711,713 within venue 710). In an example embodiment, video storage and/orvideo analysis functionality may be deployed at the venue for efficiencypurposes within devices 703 and 713.

Central Controller 760 of the proximity recognition system PRS receivesinformation from each of the proximity recognition device PRDsconfigured to send information to Central Controller 760.

In various embodiments, each PRO can send information to one or aplurality of central controller instances 760 for redundancy orinformation partitioning reasons.

With reference to FIG. 8, the architecture of a typical embodiment ofcentral controller 800 of the enhanced video system is depicted inaccordance with the definitions provided above.

Central controller 800 includes one or more central processing units(CPU) 801 to execute instructions which are delivered or installedelectronically (software) to central controller 800 such as a serverprogram to manage the operation of system. Primary storage mechanism 820is coupled to CPU 801 by interface 822 and is used to temporarily storeinstructions as well as to input and output data. CPU or CPU complex 801is also coupled by interface 823 to other secondary storage medium 821which is used to store information permanently even when centralcontroller 800 is not powered. Information can include instructions andrelevant information such as operational state data as well asconfiguration parameters.

For the purposes of system administration including system activity andstatus review, capacity optimization, or system configuration amongother functions, graphic user interface (GUI) 811 of some form isoptionally provided that connects with CPU 801 directly via localconnectivity 804 or optionally via Network Interface 802. Optionally,Resource Manager 810 is connected to CPU 801 directly or via localconnectivity 803 or optionally via Network interface 802. ExemplaryResource Manager 810 entities that are commercially available includeSplunk Enterprise and Hewlett Packard's Network Management Centerproduct.

CPU complex 801 is also coupled by interface 805 to databases used topersistently store information about the status of the proximityrecognition system PRS overall. Database 831 stores information aboutthe venues registered with central controller 800 including some optimalcombination of their name, contact information, security credentials,street address, global address expressed in latitude and longitude andpossible site specific information. Database 832 stores informationabout the proximity recognition devices (PRDs) known to centralcontroller 800 including some optimal combination of their name,communications and/or IP address, assigned venue, location within thevenue, previously assigned venues, contact information, securitycredentials, and possible biometric information. Database 833 storesinformation about the mobile devices known to the instance of centralcontroller 800 including some optimal combination of device identifier,venue appearance history as well as other possible device specificanalytics information. Database 834 stores information about usersregistered with this instance of central controller 800 including name,user name, email address, company, venue access list, PRD access list,operational privilege list, account maintenance information, biometricsinformation, audit trail and possible security credentials. Database 835stores information about analytics information awarded including someoptimal combination of their venue summarization, device summarization,time of day, week, or month summarization, other historical datasummarization or other forms of analytical calculation, date, time,customer identifier, merchant identifier, third party beneficiaryidentifier, transaction identifier, and possible security credentials.Database 836 stores information about the video capture devices (VCDs)known to central controller 800 including image information received theVCDs as well as some optimal combination of their name, communicationsand/or IP address, assigned venue, location within the venue, vieworientation, view angle and field of view parameters, previouslyassigned venues, contact information, security credentials, and possiblebiometric information. Location information in databases 832 and 836would use the same coordinate system.

Databases 831, 832, 833, 834, 835, and 836 and other Secondary StorageMedium 821 are connected and configured for optimal systems operation ina manner well known to those skilled in the area of information systemsand database technology.

Central controller 800 and in particular CPU 801, is also coupled viainterface 806 to communications Network interface 802 to communicationsnetwork 750 as in FIG. 7 in a manner well known to those skilled in thearea of information systems and communications technology.

With reference to FIG. 9, an alternative architecture of a typicalembodiment of the enhanced video system is depicted.

In this embodiment, VCDs 901, 902, 903, 904, 905 and PROs 910, 911, 912,913, deployed at one or more venues 900, report to a Central controller960. The PRD and VCD elements at depicted venue 900 report to theCentral Controller 960 over some form of communications network 940using pre-existing venue communications infrastructure 920 or newcommunications infrastructure for the EVS. This venue communicationsinfrastructure is connected to communications network 940 using someform of wired or wireless such as cellular communications.

The Central Controller 960 is attached to the communications network 940using some form of communications interface 950. The Central Controller960 of the Enhanced Video System (EVS) in this example embodimentincludes a Video Management Subsystem (VMS) 970, a Proximity RecognitionSubsystem (PRS) 980, and a Cross Reference Subsystem (CRS) 990 as wellas a user interface module 991.

The VMS subsystem 970 received and processes video image informationfrom the VCDs 901, 902, 903, 904, 905. The VMS subsystem has variousdatabases 971, 972 attached. These databases enable storage, analysis,and management of image information as well as analytics informationderived from the received image information. The VMS subsystem 970, inthis example embodiment, is designed to operate in a standalone fashionand may be deployed in advance of other EVS subsystems.

The PRS subsystem 980 received and processes presence information fromPRDs 910, 911, 912, 9134 associated with wirelessly communicating mobiledevices arriving at, moving through, and leaving venue 900. The PRSsubsystem 980 has various databases 981, 982 attached. These databasesenable storage, analysis, and management of mobile device presenceinformation as well as mobile device location information derived fromreceived mobile device presence information. The PRS subsystem 980, inthis example embodiment, is designed to operate in a standalone fashionand may be deployed in advance of other EVS subsystems.

The CRS subsystem 990 interacts with VMS and the PRS subsystem 980 overestablished application programming interfaces (APIs) to receive and/orretrieve VCD and associated image information from the VMS and toreceive and/or retrieve PRD and associated mobile device presence andlocation information from the PRS.

The CRS subsystem 990 utilizes VCD and PRD location information as wellas VCD orientation and other field of view information to enable crossreferencing of three-dimensional location information derived by the PRSsubsystem for the mobile devices in the venue with the three dimensionalfields of view of VCDs deployed in the venue using well knownmathematical algorithms as previously described. Various embodiments ofthis cross referencing task are possible. In one envisioned embodiment,the relatively static nature of PRO and VCD deployments would beexploited to do a substantial amount of pre-calculation to enable theefficient cross reference analysis of image and associated field of viewto mobile devices determined to be in the field of view at the image'sspecific instant in time and vice versa. Once this cross referenceanalysis is completed, the behavior of the mobile devices of interest(DoIs) can determined using information stored in database 835 dependingon the embodiment of EVS. Once this augmentation data is gathered, thespecific image can be augmented to provide an enhanced understanding ofthe users associated with the mobile devices of interest. This imageaugmentation data can include information regarding frequency of visit,time of arrival, time on site, entrance of arrival, other informationregarding areas of the venue visited on this or previous visits to thevenue, determined association with other mobile devices during this orprevious visits, or some calculated score associated with threat levelor other measure of interest possibly using information external to theEVS.

With reference to FIG. 10, a flowchart describing the steps performed bythe proximity recognition device PRO upon system start is depicted. Invarious embodiments, the PRD starts and becomes fully operational whenpower is applied or when certain time parameters are met (such as timeof day, for example). Processing starts at step S10-1 and immediatelyproceeds to Step S10-2 in which the PRD establishes a connection withCentral Controller.

Once connectivity has been established with Central Controller, certaininformation including time synchronization, is established at StepS10-3.

Proceeding to Step S10-4, the PRO waits to receive a wireless protocoldata unit (PDU) using an antenna and wireless transceiver fromwirelessly communicating mobile devices in the vicinity of the PRD.

When a PDU has been received at Step S10-5, the PRD proceeds to StepS10-6 in which the PDU is processed according to certain rules andinstructions that have been delivered to the PRD. An example embodimentof PDU processing is described in FIG. 8.

After processing the received PDU, the PRO proceeds to Step S10-8 wherethe status of the PRD's synchronization with Central Controller ischecked. If the PRD is synchronized with Central Controller, it returnsto Step S10-4 and waits for another PDU to arrive from mobile deviceswithin range of the PRO's antenna and wireless transceiver.

At Step S10-8, if the PRD determines it is not synchronized with thecentral controller, the PRD proceeds to Step S10-7 where the PRDattempts to re-establish connection and synchronization with CentralController.

In view of the foregoing discussions pertaining to the flowchartillustrated in FIG. 10, it is understood that such a system enablesvenue operators to better understand the behavior of venue visitors,customers and potential customers equipped with wirelessly communicatingmobile devices in new ways not heretofore possible.

While this invention is described in an embodiment with reference toWiFi, the principles of this invention are easily applicable (by anaverage person skilled in the art) to other, short range communicationsprotocols (including, but not limited to, Bluetooth (IEEE802.15.1-2002/2005), Active RFID, WiMax, LTE (Long Term Evolution).

With reference to FIG. 11, a is a logic diagram illustrating the processof an EVS using information received from associated proximityrecognition devices (PRDs) operating in a venue in conjunction withimage information received from the VCDs deployed within the venue tocalculation image augmentation data.

In various embodiments, the process starts at Step S11-1 when theCentral Controller 800 starts and then proceeds to Step S11-2 to wait toreceive information from a one or more PROs regarding the presence of amobile device detected within the venue as a result of a communicationssignal being received.

When a communication signal is received and the PRO reports this to theCentral Controller 800, 960, the EVS proceeds to Step S11-3. In thisStep, the EVS determines if location data is available for theidentified mobile device. Various embodiments exist for thisdetermination. Depending on the embodiment, standard trilateration,triangulation, time of flight, or some combination of these or otherlocation calculation techniques may be utilized to determine thelocation of the mobile device with the venue. Typically, depending onthe location calculation technique, the communications signal from themobile device must be detected by three or more PRDs deployed in thevenue.

If location information is not available for the mobile deviceassociated with the received communications signal, then EVS processreturns to the Step S11-2 to await additional mobile device presencereports. If location information is available for the mobile deviceassociated with the received communications signal, then EVS proceeds toStep S11-4. In Step 11-4, the EVS determines if the location of theassociated mobile device is covered by one or more VCDs. Variousmathematic techniques are well known for this cross referencing oflocation and may be used to accomplish this task.

If overlapping VCD information is not available for the location of themobile device associated with the received communications signal, thenEVS process returns to the Step S11-2 to await additional mobile devicepresence reports. If overlapping VCD coverage is determined to beavailable for the mobile device associated with the receivedcommunications signal, then EVS proceeds to Step S11-5. In Step 11-5,the EVS determines if the associated mobile device is known to the EVS.

If the EVS views the associated mobile device as known, then it proceedsto Step S11-6 retrieve its history within the venue as well as anyassociated venues. This history can include information about previousvenue visit history associated with the mobile device, arrival time,dwell time at the venue during this and/or previous visits, possiblemovement within a group of people, group detail such as size of group aswell as information about the mobile device's path through the venueduring this and/or previous visits.

Regardless of the availability of history information for associatedmobile device, the EVS in this scenario proceeds to calculate imageaugmentation data as described previously.

With reference to FIG. 12, a is a logic diagram illustrating the processof an EVS processing user or external system request for a list ofidentifiers for mobile device determined to be in a given VCD's field ofview at a specific instant in time.

In this process, the Central Controller of the EVS is waiting atStep12-1 for requests.

In various embodiments, the process starts at Step S12-2 when theCentral Controller 800 receives a request for “In View” ie. in the fieldof view of a given VCD at a specific instant in time.

The process proceeds to Step S12-3, when the EVS determines if it hasrequired VCD field of view information to service the received request.

If the EVS does not have the required VCD field of view information toservice the received request, processing returns to Step S12-1 to awaitanother request. In the event the EVS does have the required VCD fieldof view information to service the received request, processingcontinues to Step S12-4 where the field of view of the associated VCD isconverted into a two or three dimensional based polygon. Variousembodiments of this step exist. As previously indicated, the relativelystatic nature of VCD deployments can be exploited to do a substantialamount of pre-calculation of this two or three dimensionally basedpolygon.

The process then proceeds to Step12-5, where a determination is maderegarding the availability of mobile device location information basedon PRD reporting. Various algorithms exist for cross referencing ordetermining the two or three-dimensional intersection of the polygonrepresenting the VCD of interest's field of view and with the PRDcoverage within the venue as previously described, as approximated bypolygon(s).

If PRO information is not available for the specified VCD's field ofview, then processing returns to Step S12-1. If PRD information isavailable for the specified VCD's field of view, then for thisenvisioned process of EVS, the list of mobile devices known to be in thefield of view of the VCD of interest is produced and made available tothe requesting entity.

With reference to FIG. 13, a sample operation scenario illustrating theprocess of an EVS processing a user or external system request todetermine the venue visiting (location) history and related video scenesor fields of view of the specific mobile device(s).

An external request is received by interface module 991 for a list ofidentifiers for mobile device with a parameter set (S13-1).

Cross Reference Subsystem (CRS) 990 determines the list of mobiledevices to be in a VCD's field of view (@Location A) at Time t1 asdescribed in FIG. 12 (S13-2).

For each identified mobile devices, CRS 900 retrieves its venue visithistory with timeline and corresponding location(s) and videoscene(s)/VCD field(s) of view. (S13-3)

Another external request is received by interface module 991 for a listof identifiers for mobile device with a parameter set (S13-4).

CRS 990 determines the list of mobile devices in a VCD's field of view(@Location B) at Time t2 as described in FIG. 12 (S13-5).

For each identified mobile device, CRS 900 retrieves its venue visithistory with timeline and corresponding location(s) and videoscene(s)/VCD field(s) of view (S13-6)

Another external request is received by interface module 991 to find thecommon mobile device list between the previous requests (eg. S13-2 andS13-5). S13-7

CRC 900 determines the list of mobile devices that are common(intersecting) between the previous two requests (S13-2 and S13-5).(S13-8)

For each identified mobile device, CRS 900 retrieves its venue visithistory with timeline and corresponding location(s) and videoscene(s)/VCD field(s) of view. For example, the EVS can provide a videoscene (field of view) that includes the specific wireless device@Location D at Time t10. (S13-9)

For some applications, the objective is to have a small number of commonmobile devices (eg. down to 1 specific mobile device). Based on theresult from S13-8, another external request may be received by interfacemodule 991 for a list of identifiers for mobile device with anotherparameter set (S13-10, S13-11, S13-12).

And then another external request is received by interface module 991 tofind the common mobile device list between the previous requests (eg.S13-2, S13-5, and S13-11). (S13-7)

CRC 900 determines the list of mobile devices that is common(intersecting) between the previous three requests (S13-2, S13-5, andS13-11). (S13-8)

The process of S13-10, S13-11, S13-12 follows by S13-7, S13-8, S13-9 maybe carried out in multiple iterations until the desired mobile device isdetermined. In S13-7, different list of mobile devices may be requestedby the user, eg. Intersect of S13-2 and S13-5 or intersect of S13-2,S13-5, S13-11.

With reference to FIG. 4, a sample collection of video image searchoutput is shown with example information specifying the location of thescene or VCD where the image was captured along with the time the imagewas captured as well as information regarding the one or more objects inthe scene of the VCD field of view. This example collection of imagesand associated information could be generated by the EVS given an imageof interest. This object information is referred to in the FIG. 4 as“Pattern”. This pattern information for example could indicate theappearance in the scene of a single person, with male sex, approximateheight of six feet, wearing purple pants. More sophisticated examples ofobject information associated with overall image information are wellknown.

Based on this object information and associated “patterns”, either ahuman or a computer could determine high probability matches betweenimages based on a commonality of patterns through various techniques andalgorithms.

These high probability matches could be used to determine mobile devicecommonality as described in FIG. 4 to increase or decrease theconfidence of the match of the image with the original image ofinterest. This process can be repeated across all images identified bythe video image search process to increase or decrease the confidence ofthe match with the original image of interest. Through some algorithmthis confidence could be used to modify the video search output orresults associated with the original image of interest.

Accordingly, while this invention has been described with reference tothe illustrative embodiments, this description is not intended to beconstrued in any limiting sense. Various modifications of theillustrative embodiments, as well as other embodiments of the invention,will be apparent to persons skilled in the art upon reference to thisdescription. It is therefore contemplated that any future patent claimswill cover any such modifications or embodiments which falls within thescope of the invention. In particular, regardless of how the locationinformation of visiting wireless mobile devices was obtained, theprinciples of this invention are applicable. What this invention teachesis that enhancement of video systems using wireless device proximitydetection. The proximity of a device communicating wirelessly is sensedand cross referenced with received video image information. Throughtime, movement of wirelessly communicating mobile devices through avenue or set of venues can be deduced and additionally cross referencedto and augmented over image data from the set of video capture devices.

The invention claimed is:
 1. One or more tangible, non-transitory,machine-readable media storing instructions that when executed by one ormore processors effectuate operations comprising: determining, with acomputer system, that a first mobile device is present at a firstlocation in a venue based on a signal of the first mobile devicereceived by a proximity recognition device at a first time point,wherein the proximity recognition device is configured to observe, usingan antenna of and wireless transceiver of the proximity recognitiondevice, communication signals between wirelessly communicating mobiledevices and cause, based on the communication signals, locations of thewirelessly communicating mobile devices to be determined; selecting,with the computer system, a first image captured by a first cameraassociated with the first location, wherein a field of view of the firstcamera spatially overlaps with an antenna range of the proximityrecognition device, and wherein the first camera is one of a pluralityof cameras; detecting, with the computer system, a first person depictedin the first image based on both the first image and the first locationdetermined based on the signal of the first mobile device received bythe proximity recognition device; associating, with the computer system,in response to detecting the first person, the first person with thefirst mobile device; determining, with the computer system, a first setof visual attributes associated with the first person based on the firstimage, wherein the first set of visual attributes is indicative of afirst height of the first person and a first color visually associatedwith the first person; detecting, with the computer system, a secondperson positioned at a second location in the venue at a second timepoint based on a second image captured by a second camera of theplurality of cameras; determining, with the computer system, a secondset of visual attributes associated with the second person based on thesecond image, wherein the second set of visual attributes is indicativeof a second height of the second person and a second color visuallyassociated with the second person; determining, with the computersystem, a match probability based on the first set of visual attributesand the second set of visual attributes; determining, with the computersystem, whether the match probability satisfies a confidence threshold;and in response to a determination that the match probability satisfiesthe confidence threshold, storing, with the computer system, anindication of a match between the second person and the first person inmemory.
 2. The one or more media of claim 1, wherein the proximityrecognition device is a first proximity recognition device, and whereinthe operations further comprise: determining that the first mobiledevice is present at the second location based on a second signal of thefirst mobile device received b y a second proximity recognition deviceat the second time point; determining a device path based on the firstlocation, the first time point, the second location, and the second timepoint, wherein the device path includes a first position representing anarrival into the venue and a second position representing a departurefrom the venue; and visually depicting the device path on a displaysystem.
 3. The one or more media of claim 1, the operations furthercomprising: determining a two-dimensional shape based on the field ofview; determining whether the two-dimensional shape spatially overlapswith the first location; and associating the first camera with the firstlocation based on a determination that the two-dimensional shapespatially overlaps with the first location.
 4. The one or more media ofclaim 1, the operations further comprising: determining athree-dimensional shape based on the field of view; determining whetherthe three-dimensional shape spatially overlaps with the first locationusing a three-dimensional polygonal intersection algorithm; andassociating the first camera with the first location based on adetermination that the three-dimensional shape spatially overlap withthe first location.
 5. The one or more media of claim 1, wherein thematch probability is a first match probability, the operations furthercomprising: detecting a third person at the first time point based on athird image captured by a third camera of the plurality of cameras;determining a third set of visual attributes associated with the thirdperson based on the third image, wherein the third set of visualattributes is indicative of a third height of the third person and athird color visually associated with the third person; determining asecond match probability based on the first set of visual attributes andthe third set of visual attributes; determining whether the second matchprobability satisfies the confidence threshold; in response to adetermination that the second match probability satisfies the confidencethreshold, associating the third person with the first person and thesecond person.
 6. The one or more media of claim 1, the operationsfurther comprising causing a display system to display an augmentedversion of the second image, wherein the augmented version of the secondimage comprises a visual display of the first time point.
 7. The one ormore media of claim 1, wherein the proximity recognition device is afirst proximity recognition device, and wherein the first proximityrecognition device is located at a first proximity device position, andwherein the operations further comprise: detecting the first mobiledevice with a second proximity recognition device at the first timepoint, wherein the second proximity recognition device is located at asecond proximity device position; detecting the first mobile device witha third proximity recognition device at the first time point, whereinthe third proximity recognition device is located at a third proximitydevice position; and determining the first location based on the firstproximity device position, the second proximity device position, and thethird proximity device position.
 8. The one or more media of claim 1,the operations further comprising: receiving a device orientation of thefirst camera and a device position of the first camera; and determiningthe field of view of the first camera based on the device orientationand the device position.
 9. The one or more media of claim 1, theoperations further comprising: determining that the first mobile deviceis present at a third location at a third time point using an additionalproximity recognition device; capturing a third image of the thirdlocation using the second camera; determining that a person associatedwith the first mobile device is not captured by the third image due toan obstruction; and determining a device path, wherein the device pathcomprises the first location, the first time point, the second location,the second time point, the third location, and the third time point. 10.The one or more media of claim 1, the operations further comprising:determining an association set of visual attributes between the firstmobile device and a second mobile device based on the first image;detecting the second mobile device at a third location at a third timepoint using a second proximity recognition device, wherein the firstmobile device is not detected at the third location at the third timepoint; and determining a device path, wherein the device path comprisesthe first location, the first time point, the second location, thesecond time point, the third location, and the third time point.
 11. Theone or more media of claim 1, the operations further comprising:determining whether the first mobile device is in a list of knowndevices; and in response to a determination that the first mobile deviceis in the list of known devices, retrieving a history of the firstmobile device; and visually display a value of the history of the firstmobile device.
 12. The one or more media of claim 11, wherein: thehistory of the first mobile device comprises a count of a group ofpeople, wherein the first mobile device was determined to have movedwithin the group of people in a previous visit; and visually displayingthe value of the history of the first mobile device comprises visuallydisplaying the count of the group of people.
 13. The one or more mediaof claim 1, wherein the signal is a wireless local area network signal,IEEE 802.15.1-2002/2005 signal, Active RFID signal, WiMax signal, orLong Term Evolution signal.
 14. The one or more media of claim 1,wherein the field of view is a first field of view, and wherein theoperations further comprise: determining a first set of mobile devicesdetected at the first location, wherein the first set of mobile devicescomprises the first mobile device; determining a second set of mobiledevices detected at the second location, wherein the second set ofmobile devices comprises the first mobile device; and determining athird set of mobile devices based on which devices are in both the firstset of mobile devices and the second set of mobile devices, wherein thethird set of mobile devices comprises the first mobile device.
 15. Theone or more media of claim 1, wherein: the first mobile device is one ofa plurality of mobile devices; the signal is one of a first plurality ofsignals; the first person is one of a plurality of physically differentpersons; and the operations further comprise: determining that theplurality of mobile devices is present at the first location based onthe first plurality of signals received by the proximity recognitiondevice at the first time point; detecting the plurality of physicallydifferent persons depicted in the first image based on the first image,wherein the plurality of physically different persons is determined tobe a group; detecting that the plurality of mobile devices is at thesecond location based a second set of signals received by a secondproximity recognition device at the second time point; and generating analert indicating that the group has arrived at the venue.
 16. The one ormore media of claim 1, further comprising determining the first locationbased on a trilateration computation, a triangulation computation, or atime of flight computation.
 17. The one or more media of claim 1,wherein the first image is associated with a set of values defining thefield of view captured by the first image.
 18. The one or more media ofclaim 1, the operations further comprising displaying a future timepoint associated with a future visit of the first mobile device to thevenue, wherein the future time point occurs after the first mobiledevice is determined to have departed from the venue.
 19. The one ormore media of claim 1, wherein determining that the first mobile deviceis present at the first location comprises: changing a signal strengththreshold of the proximity recognition device; determining whether asignal strength of the signal satisfies the signal strength threshold;and in response to a determination that the signal strength satisfiesthe signal strength threshold, determine the first mobile device ispresent at the first location.
 20. A method comprising: determining,with a computer system, that a first mobile device is present at a firstlocation in a venue based on a signal of the first mobile devicereceived by a proximity recognition device at a first time point,wherein the proximity recognition device is configured to observe, usingan antenna of and wireless transceiver of the proximity recognitiondevice, communication signals between wirelessly communicating mobiledevices and cause, based on the communication signals, locations of thewirelessly communicating mobile devices to be determined; selecting,with the computer system, a first image captured by a first cameraassociated with the first location, wherein a field of view of the firstcamera spatially overlaps with an antenna range of the proximityrecognition device, and wherein the first camera is one of a pluralityof cameras; detecting, with the computer system, a first person depictedin the first image based on both the first image and the first locationdetermined based on the signal of the first mobile device received bythe proximity recognition device; associating, with the computer system,in response to detecting the first person, the first person with thefirst mobile device; determining, with the computer system, a first setof visual attributes associated with the first person based on the firstimage, wherein the first set of visual attributes is indicative of afirst height of the first person and a first color visually associatedwith the first person; detecting, with the computer system, a secondperson positioned at a second location in the venue at a second timepoint based on a second image captured by a second camera of theplurality of cameras; determining, with the computer system, a secondset of visual attributes associated with the second person based on thesecond image, wherein the second set of visual attributes is indicativeof a second height of the second person and a second color visuallyassociated with the second person; determining, with the computersystem, a match probability based on the first set of visual attributesand the second set of visual attributes; determining, with the computersystem, whether the match probability satisfies a confidence threshold;and in response to a determination that the match probability satisfiesthe confidence threshold, storing, with the computer system, anindication of a match between the second person and the first person inmemory.
 21. The method of claim 20, wherein the proximity recognitiondevice is a first proximity recognition device, and further comprising:determining that the first mobile device is present at the secondlocation based on a second signal of the first mobile device received bya second proximity recognition device at the second time point;determining a device path based on the first location, the first timepoint, the second location, and the second time point, wherein thedevice path includes a first position representing an arrival into thevenue and a second position representing a departure from the venue; andvisually depicting the device path on a display system.
 22. The methodof claim 20, further comprising: determining a two-dimensional shapebased on the field of view; determining whether the two-dimensionalshape spatially overlaps with the first location; and associating thefirst camera with the first location based on a determination that thetwo-dimensional shape spatially overlaps with the first location. 23.The method of claim 20, further comprising: determining athree-dimensional shape based on the field of view; determining whetherthe three-dimensional shape spatially overlaps with the first locationusing a three-dimensional polygonal intersection algorithm; andassociating the first camera with the first location based on adetermination that the three-dimensional shape spatially overlap withthe first location.
 24. The method of claim 20, wherein the matchprobability is a first match probability, and further comprising:detecting a third person at the first time point based on a third imagecaptured by a third camera of the plurality of cameras; determining athird set of visual attributes associated with the third person based onthe third image, wherein the third set of visual attributes isindicative of a third height of the third person and a third colorvisually associated with the third person; determining a second matchprobability based on the first set of visual attributes and the thirdset of visual attributes; determining whether the second matchprobability satisfies the confidence threshold; in response to adetermination that the second match probability satisfies the confidencethreshold, associating the third person with the first person and thesecond person.
 25. The method of claim 20, further comprising: causing adisplay system to display an augmented version of the second image,wherein the augmented version of the second image comprises a visualdisplay of the first time point.
 26. The method of claim 20, wherein theproximity recognition device is a first proximity recognition device,and wherein the first proximity recognition device is located at a firstproximity device position, and further comprising: detecting the firstmobile device with a second proximity recognition device at the firsttime point, wherein the second proximity recognition device is locatedat a second proximity device position; detecting the first mobile devicewith a third proximity recognition device at the first time point,wherein the third proximity recognition device is located at a thirdproximity device position; and determining the first location based onthe first proximity device position, the second proximity deviceposition, and the third proximity device position.
 27. The method ofclaim 20, further comprising: receiving a device orientation of thefirst camera and a device position of the first camera; and determiningthe field of view of the first camera based on the device orientationand the device position.
 28. The method of claim 20, further comprising:determining that the first mobile device is present at a third locationat a third time point using an additional proximity recognition device;capturing a third image of the third location using the second camera;determining that a person associated with the first mobile device is notcaptured by the third image due to an obstruction; and determining adevice path, wherein the device path comprises the first location, thefirst time point, the second location, the second time point, the thirdlocation, and the third time point.
 29. The method of claim 20, furthercomprising: determining an association set of visual attributes betweenthe first mobile device and a second mobile device based on the firstimage; detecting the second mobile device at a third location at a thirdtime point using a second proximity recognition device, wherein thefirst mobile device is not detected at the third location at the thirdtime point; and determining a device path, wherein the device pathcomprises the first location, the first time point, the second location,the second time point, the third location, and the third time point. 30.The method of claim 20, further comprising: determining whether thefirst mobile device is in a list of known devices; and in response to adetermination that the first mobile device is in the list of knowndevices, retrieving a history of the first mobile device; and visuallydisplay a value of the history of the first mobile device.
 31. Themethod of claim 30, wherein: the history of the first mobile devicecomprises a count of a group of people, wherein the first mobile devicewas determined to have moved within the group of people in a previousvisit; and visually displaying the value of the history of the firstmobile device comprises visually displaying the count of the group ofpeople.
 32. The method of claim 20, wherein the signal is a wirelesslocal area network signal, IEEE 802.15.1-2002/2005 signal, Active RFIDsignal, WiMax signal, or Long Term Evolution signal.
 33. The method ofclaim 20, wherein the field of view is a first field of view, andfurther comprising: determining a first set of mobile devices detectedat the first location, wherein the first set of mobile devices comprisesthe first mobile device; determining a second set of mobile devicesdetected at the second location, wherein the second set of mobiledevices comprises the first mobile device; and determining a third setof mobile devices based on which devices are in both the first set ofmobile devices and the second set of mobile devices, wherein the thirdset of mobile devices comprises the first mobile device.
 34. The methodof claim 20, wherein: the first mobile device is one of a plurality ofmobile devices; the signal is one of a first plurality of signals; thefirst person is one of a plurality of physically different persons; andfurther comprising: determining that the plurality of mobile devices ispresent at the first location based on the first plurality of signalsreceived by the proximity recognition device at the first time point;detecting the plurality of physically different persons depicted in thefirst image based on the first image, wherein the plurality ofphysically different persons is determined to be a group; detecting thatthe plurality of mobile devices is at the second location based a secondset of signals received by a second proximity recognition device at thesecond time point; and generating an alert indicating that the group hasarrived at the venue.
 35. The method of claim 20, further comprisingdetermining the first location based on a trilateration computation, atriangulation computation, or a time of flight computation.
 36. Themethod of claim 20, wherein the first image is associated with a set ofvalues defining the field of view captured by the first image.
 37. Themethod of claim 20, further comprising: displaying a future time pointassociated with a future visit of the first mobile device to the venue,wherein the future time point occurs after the first mobile device isdetermined to have departed from the venue.
 38. The method of claim 20,wherein determining that the first mobile device is present at the firstlocation comprises: changing a signal strength threshold of theproximity recognition device; determining whether a signal strength ofthe signal satisfies the signal strength threshold; and in response to adetermination that the signal strength satisfies the signal strengththreshold, determine the first mobile device is present at the firstlocation.